Debugging of Markov Decision Processes (MDPs) Models

Accelerated decomposition techniques for large discounted Markov decision processes

Many hierarchical techniques to solve large Markov decision processes (MDPs) are based on the partition of the state space into strongly connected components (SCCs) that can be classified into some levels. In each level, smaller problems named restricted MDPs are solved, and then these partial solutions are combined to obtain the global solution. In this paper, we first propose a novel algorith...

Counterexamples for Model Checking of Markov Decision Processes

The debugging of stochastic system models relies on the availability of diagnostic information. Classic probabilistic model checkers, which are based on iterated numerical probability matrix operations, do not provide such diagnostic information. In precursory work, we have devised counterexample generation methods for continuousand discrete-time Markov Chains based on heuristics guided explici...

MDPs Semi - Markov decision processes Hidden Markov models Partially observable SMDPs Hierarchical HMMs

Markov decision processes: a tool for sequential decision making under uncertainty.

We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment proble...

Ordinal Decision Models for Markov Decision Processes

Setting the values of rewards in Markov decision processes (MDP) may be a difficult task. In this paper, we consider two ordinal decision models for MDPs where only an order is known over rewards. The first one, which has been proposed recently in MDPs [23], defines preferences with respect to a reference point. The second model, which can been viewed as the dual approach of the first one, is b...